Method and apparatus for video interpretation of carotid intima-media thickness

ABSTRACT

A system for automatically determining a thickness of a wall of an artery of a subject includes an ECG monitoring device that captures an electrocardiogram (ECG) signal from the subject, and an ultrasound video imaging device, coupled to the ECG monitoring device, that receives the ECG signal from the ECG monitoring device, and captures a corresponding ultrasound video of the wall of the artery of the subject. The system produces a plurality of frames of video comprising the ultrasound video of the wall of the artery of the subject and an image of the ECG signal. A processor is configured to select a subset of the plurality of frames of the ultrasound video based on the image of the (ECG) signal, locate automatically a region of interest (ROI) in each frame of the subset of the plurality of frames of the video using a machine-based artificial neural network and measure automatically a thickness of the wall of the artery in each ROI using the machine-based artificial neural network.

BACKGROUND

The subject matter described herein relates to systems and methods fordetermining carotid artery intima-media thickness (CIMT).

Cardiovascular disease (CVD) is the number one killer in the UnitedStates. Nevertheless, CVD is largely preventable. However, the key is toidentify at-risk persons before coronary events occur, so thatpreventive care can be prescribed appropriately. A noninvasiveultrasonography method that has proven to be valuable for predictingindividual CVD risk involves determining a person's carotid arteryintima-media thickness (CIMT). Interpretation of CIMT ultrasonographicvideos involves three manual operations: 1) selection of end-diastolicultrasonographic frames (EUFs) in each video; 2) localization of aregion of interest (ROI) in each selected EUF; and 3) identification ofthe intima-media boundaries within each ROI to measure CIMT. Withreference to FIG. 1, which illustrates a longitudinal view of the commoncarotid artery of a human subject in an ultrasonographic B-scan image100, CIMT is defined as the distance between the lumen-intima interfaceand the media-adventitia interface, measured approximately 1 cm from thecarotid bulb on the far wall of the common carotid artery at the end ofthe diastole. Therefore, interpretation of a CIMT video involves 3operations: 1) select 3 EUFs in each video (the cardiac cycle indicatorshows to where in the cardiac cycle the current frame in the videocorresponds); 2) localize an ROI approximately 1 cm distal from thecarotid bulb in the selected EUF; and 3) measure the CIMT within thelocalized ROI.

These three operations, and in particular, the third step of CIMTmeasurement, are not only tedious and laborious but also subjective tolarge inter-operator variability if guidelines are not properlyfollowed. These factors have hindered the widespread utilization of CIMTin clinical practice. To overcome this limitation, what is needed is anew system to accelerate CIMT video interpretation through automation ofthe operations in a novel, unified framework using machine-basedartificial neural networks such as convolutional neural networks (CNNs).

SUMMARY

Embodiments of the invention relate to systems, methods and program codefor automatically determining a thickness of a wall of an artery of asubject. In one embodiment, an ECG monitoring device captures anelectrocardiogram (ECG) signal from the subject, and an ultrasound videoimaging device, coupled to the ECG monitoring device, receives the ECGsignal from the ECG monitoring device, and also captures a correspondingultrasound video of the wall of the artery of the subject. Theultrasound video imaging device produces a plurality of frames of videocomprising the ultrasound video of the wall of the artery of the subjectand, in one embodiment, an image of the ECG signal is integrated in, ordisplayed on, the frames. A processor coupled to the ultrasoundvideo-imaging device is configured via executable computer program codeto select a subset of the plurality of frames of the ultrasound videobased on the image of the (ECG) signal. The processor is furtherconfigured via the executable computer program code to locateautomatically a region of interest (ROI) in each frame of the subset ofthe plurality of frames of the video and measure automatically athickness of the wall of the artery in each ROI using a machine-basedartificial neural network.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a longitudinal view of the common carotid artery of ahuman subject in an ultrasonographic B-scan image.

FIG. 2 illustrates a system for implementing embodiments of theinvention.

FIG. 3 illustrates a flow chart of a process according to embodiments ofthe invention.

FIG. 4 illustrates accumulated difference images according to anembodiment of the invention.

FIG. 5 shows a frame selection system for a test video, according to oneembodiment of the invention.

FIG. 6 illustrates aspects of embodiments of the invention related tolocating a region of interest (ROI) in each frame of a subset of framesof video.

FIG. 7 illustrates aspects of embodiments of the invention related tolocating a region of interest (ROI) in each frame of a subset of framesof video.

FIG. 8 illustrates aspects of embodiments of the invention related tomeasuring a thickness of the wall of the artery in a region of interest(ROI).

FIG. 9 illustrates aspects of embodiments of the invention related tomeasuring a thickness of the wall of the artery in a region of interest(ROI).

DETAILED DESCRIPTION

Embodiments of the invention relate to a system for automaticallydetermining a thickness of a wall of an artery of a subject. The system,as shown in FIG. 2, and with reference to the process shown in theflowchart in FIG. 3, comprises an ECG monitoring device 280 thatcaptures at 305 an electrocardiogram (ECG) signal from the subject. Thesystem further includes an ultrasound video imaging device 220, coupledto the ECG monitoring device 280, in one embodiment, by way of computingdevice 210, that receives the ECG signal from the ECG monitoring device,and captures at 310 a corresponding ultrasound video of the wall of theartery of the subject. In one embodiment, the ultrasound video imagingdevice outputs at 315 a plurality of frames of video comprising theultrasound video of the wall of the artery of the subject and an imageof the ECG signal.

In another embodiment, the ECG signal and the ultrasound video arereceived by a computing device 210 coupled to the ECG monitoring deviceand ultrasound video imaging device via communication link 230, and aprocessor 240 combines, stores in storage device 250, and outputs at 315to display device 270, video frames combining the ultrasound video andan image of the ECG signal.

In one embodiment, the obtained ECG signals are separately encoded inthe ultrasound video images in a file, such as a Digital Imaging andCommunications in Medicine (DICOM) image file, and they are alsosynchronized with corresponding frame numbers for the ultrasound videoimages. (Note: DICOM is also known as NEMA (National ElectricalManufacturers Association) standard PS3, and as ISO standard 12052:2006“Health informatics—Digital imaging and communication in medicine(DICOM) including workflow and data management”). In one embodiment, theECG signals are reconstructed from the images when the ECG signal isoverlaid on top of, or otherwise combined with, the ultrasound images,if ECG signal was not presented via a separate channel. In anotherembodiment, the ECG signal is separately presented or available, anddoes not need to be reconstructed from ultrasound images with which itis combined. It is appreciated that some embodiments may operate withultrasound video files for which there are no means for separating theECG signal, for example, video files employing video formats such asAVI, MOV, MP4, etc. It is noted also that ECG signal encoding accordingto the DICOM standard is in fact not standardized, so each ultrasoundvideo/ECG monitor device manufacturer will use proprietary formatting tostore the ECG signals using unique DICOM tags which may present achallenge in interpreting ECG signals. Therefore, one major advantage ofembodiments of the present invention is that frame detection can extractany type of ECG signal even if signal cues are missing and/or wraparound to the left on the display screen from the embedded images whereseparate ECG signals cannot be obtained directly from DICOM video formatand without knowing a particular manufacturer's specification for ECGsignal encoding format.

A processor 240 of the computing device 210 is then configured at 320 toselect a subset of the plurality of frames of the ultrasound video basedon the image of the (ECG) signal. In one embodiment, the selection maybe accomplished using a machine-based artificial neural network, such asconvolutional neural networks. In one embodiment, the processor isconfigured to select a plurality of end-diastolic ultrasound frames(EUFs) based on corresponding R-waves in a QRS complex in the image ofthe ECG signal, using the machine-based artificial neural network. Oncethe subset of video frames is selected based on the ECG signal, theprocessor is configured at 325 to automatically localize a region ofinterest (ROI) in each frame of the subset of the plurality of frames ofthe video using the machine-based artificial neural network,

With regard to step 325, in one embodiment, the processor is configuredto estimate a location of the ROI in each frame of the subset of theplurality of frames of the video, and a location of a well-known area ofthe artery of the subject as a contextual constraint, and then refinethe estimated location of the ROI given the estimated location of thewell-known area of the artery. In one embodiment, the location of thewell-known area of the artery is the location of the carotid bulb of asubject's common carotid artery (CCA) and the estimated location of theROI is approximately 1 centimeter from the carotid bulb on a far wall ofthe subject's CCA.

Having determined the location of the ROI in each frame in the subset ofvideo frames, the system measures automatically at 330 a thickness ofthe wall of the artery in each ROI using the machine-based artificialneural network. In particular, the processor is configured to measureautomatically a carotid intima-media thickness (CIMT) of the far wall ofthe carotid artery in each ROI using the machine-based artificial neuralnetwork. This measurement is accomplished in one embodiment by firstdetecting a lumen-intima interface of the wall of the carotid artery,further detecting a media-adventitia interface of the wall of thecarotid artery, and then measuring the distance between the lumen-intimainterface and the media-adventitia interface to determine the CIMT ofthe carotid artery.

In one embodiment, CIMT examinations may be performed with highresolution B-mode ultrasonography using a 15 MHz linear array transducerwith fundamental frequency only (such as available from Acuson Sequoia,Mountain View, Calif., USA). The carotid screening protocol begins withscanning up from the lower neck in a transverse manner to the carotidartery and then further to the carotid bulb and to internal and externalcarotid arteries. The probe is then turned to obtain a longitudinal viewof the common carotid artery, as illustrated in FIG. 1. A sonographeroptimizes the 2-dimensional images of the lumen-intima andmedia-adventitia interfaces at the level of the common carotid artery byadjusting overall gain, time gain, compensation, and focus position.After the parameters are adjusted, the sonographer captures two CIMTvideos focused on the common carotid artery from at least two optimalangles of incidence. The same procedure may be repeated for the otherside of neck, resulting in a total of four CIMT videos for each patient.In one embodiment, the videos are recorded at 24 frames/second andconsist of 640×480 pixels of video resolution in the ultrasound images.The pixel spacing is 0.09 mm/pixel along both x and y directions.

Further details of embodiments of the invention that provide a unifiedsolution based on convolutional neural networks (CNNs) for automatingthe three main tasks in CIMT video interpretation, namely, frameselection, region of interest (ROI) localization, and intima-mediathickness measurement, are provided below.

Frame Selection

The first step in automatically determining a thickness of a wall of anartery of a subject involves obtaining an ultrasound video of the wallof the artery of the subject, the ultrasound video comprising aplurality of frames of video, obtaining a correspondingelectrocardiogram (ECG) signal from the subject, and then selecting asubset of the plurality of frames of the video based on thecorresponding (ECG) signal. In one embodiment, selecting the subset ofthe plurality of frames of the ultrasound video based on thecorresponding ECG signal involves selecting end-diastolic ultrasoundframes (EUFs) based on corresponding R-waves in a QRS complex of the ECGsignal. In one embodiment, selecting the subset of the plurality offrames of the ultrasound video based on a correspondingelectrocardiogram (ECG) signal involves automatically selecting thesubset of the plurality of frames of the ultrasound video based on animage of the corresponding electrocardiogram (ECG) signal displayed inthe ultrasound video, using a machine-based artificial neural network,such as a convolutional neural network.

In one embodiment, given a CIMT video, the first step in cardiovascularrisk assessment selects three EUFs. The CIMT test is routinely performedwith ECG, and an operator normally selects the three EUFs on the basisof the ECG signal that is displayed at the bottom of theultrasonographic frames. Each frame in the CIMT video corresponds to aparticular location in the printed ECG signal. To establish thiscorrespondence, as shown in FIG. 1, a black line indicator (cardiaccycle indicator (1)) is displayed on the image of the ECG signal,indicating to where in the cardiac cycle the current video framecorresponds. In routine clinical practice, the operator selects the EUFsso that the corresponding black line indicator coincides with the R wavein the QRS complex of the printed ECG signals.

One embodiment automates the frame selection process by automaticallyidentifying the frames that correspond to the R wave in the QRS complexin the image of ECG signal. According to an embodiment, the segment ofthe ECG signal that is masked by the black line indicator in the currentframe is reconstructed, and then a determination is made as to whetherthe restored wavelet (that is, the small part of the ECG signal that isreconstructed) resembles the appearance of an R wave or not. For thispurpose, and with reference to FIG. 4, accumulated difference images 405are used to capture the missing wavelets 410 and then a CNN is used toclassify these captured wavelets into R wave or non-R wave categories.

Let I^(t) denote an image sub region selected from the bottom of anultrasonographic frame 415 (e.g., the bottom 20%) that contains thedisplayed ECG signal. First, the set of difference images d^(t) areconstructed by subtracting every consecutive pair of images,dt=I^(t)−I^(t+1)|, and then form accumulated difference images by addingup every 3 neighboring difference images,

D ^(t)=Σ_(i=0) ² d ^(t−i)

An accumulated difference image D^(t) can capture the segment of the ECGsignal that is masked by the black line indicator at frame t. Second,the location of the restored wavelet in each accumulated differenceimage is determined. In one embodiment, this is accomplished by findingthe weighted centroid c=[c_(x), c_(y)] of each accumulated differenceimage D^(t) as follows:

$c = {\frac{1}{Z_{t}}{\sum\limits_{p \in D^{t}}\; {{D^{t}\left( {p_{x},p_{y}} \right)} \times p}}}$

where p=[p_(x); p_(y)] is a pixel in the accumulated difference imageand

Z ^(t)=Σ_(pεD) _(t) D ^(t)(p _(x) ,p _(y))

is a normalization factor that ensures the weighted centroid stayswithin the image boundary. After centroids are identified, patches ofsize 32×32 pixels are extracted around the centroid locations.Specifically, patches with up to 2 pixel translations from each centroidare extracted. In one embodiment, data augmentation is not performed byscaling the patches because doing so would inject label noise in thetraining set. For instance, a small, restored wavelet may take theappearance of an R wave after expanding, or an R wave may look like anon-R wave after shrinking. Rotation-based patch augmentation is alsonot performed because it is not expected for the restored wavelets toappear with rotation in the test image patches. After collection, thepatches are binarized according to the method described in N. Otsu, “Athreshold selection method from gray-level histograms,” Automatica, vol.11, no. 285-296, pp. 23-27, 1975. Each binary patch is then labeled aspositive if it corresponds to an EUF (or an R wave); otherwise, it islabeled as negative. Basically, given a patch, one embodiment initiallydetermines the accumulated difference image from which the patch isextracted, followed by tracing back to the underlying difference imagesto check whether they are related to the EUF or not. After the patchesare labeled, a stratified set is formed with 96,000 patches to train aCNN for distinguishing between R waves and non-R waves.

FIG. 5 shows a frame selection system for a test video, according to oneembodiment of the invention. An accumulated difference image is computedat 505 for each original image frame 500 in the video. Binarized imagepatches 510 are then extracted from the weighted centroids of theaccumulated difference images. At 515, the probability of each framebeing the EUF is measured as the average probabilities assigned by theCNN to the corresponding patches. By concatenating the resultingprobabilities for all frames in the video, a noisy probability signal isobtained at 520 whose local maxima indicate the locations of the EUFs.However, the generated probability signals often exhibit abrupt changes,which can cause too many local maxima along the signal. One embodiment,therefore, first smoothed at 525 the probability signal using a Gaussianfunction and then finds the EUFs by locating the local maxima of thesmoothed signals. FIG. 5, for illustration purposes, also shows at 530the reconstructed ECG signal, which is computed as the average of theaccumulated difference images,

$\frac{1}{N}{\sum\limits_{t = 1}^{N}\; D^{t}}$

with N being the number of frames in the video. As seen, the probabilityof a frame being the EUF reaches a maximum around the R waves of the QRScomplexes (as desired) and then smoothly decays as it distances from theR waves. By mapping the locations of the local maxima to the framenumbers, EUFs in the video can be identified.

ROI Localization

ROI localization involves locating automatically a region of interest(ROI) in each frame of the subset of the plurality of frames of thevideo using a machine-based artificial neural network. One embodimentfor ROI localization estimates, simultaneously, a location of the ROI ineach frame of the subset of the plurality of frames of the video, and alocation of a well-known area of the artery of the subject as acontextual constraint, and then refines the estimated location of theROI given the estimated location of the well-known area of the artery.As described above, in one embodiment, the location of the well-knownarea of the artery comprises the location of a carotid bulb of asubject's common carotid artery (CCA), and the estimated location of theROI is approximately 1 centimeter from the carotid bulb on a far wall ofthe subject's CCA.

Accurate localization of the ROI can be challenging because, as seen inFIG. 1, no notable differences can be readily observed in imageappearance among the ROIs on the far wall of the carotid artery. Toovercome this challenge, one embodiment uses the location of the carotidbulb as a contextual constraint. This constraint is chosen for tworeasons: 1) the carotid bulb appears as a distinct dark area in theultrasonographic frame and thus can be uniquely identified; and 2)according to the consensus statement of American Society ofElectrocardiography for cardiovascular risk assessment, the ROI shouldbe placed approximately 1 cm from the carotid bulb on the far wall ofthe common carotid artery. The former motivates the use of the carotidbulb location as a constraint from a technical point of view, and thelatter justifies this constraint from a clinical standpoint. See J.Stein, C. Korcarz, R. Hurst, E. Lonn, C. Kendall, E. Mohler, S. Najjar,C. Rembold, and W. Post, “American Society of Echocardiography carotidintima-media thickness task force. Use of carotid ultrasound to identifysubclinical vascular disease and evaluate cardiovascular disease risk: aconsensus statement from the American Society of Echocardiographycarotid intima-media thickness task force. Endorsed by the society forvascular medicine,” J Am Soc Echocardiogr, vol. 21, no. 2, pp. 93-111,2008.

One embodiment of the invention incorporates this constraint by traininga CNN for 3-class classification that simultaneously localizes both theROI and the carotid bulb and then refines the estimated location of theROI given the location of the carotid bulb. FIG. 6 shows how the imagepatches 605 are extracted from a training frame 610. Data augmentationmay be performed, by extracting the training patches within a circlearound the locations of the carotid bulbs and the ROIs. The backgroundpatches are extracted from a grid of points sufficiently far from thelocations of the carotid bulbs and the ROIs. Of note, the describedtranslation-based data augmentation is sufficient for this applicationgiven a database that provides a relatively large number of trainingEUFs, from which a large set of training patches can be collected. Afterthe patches are collected, a stratified training set is formed, in oneembodiment, with approximately 410,000 patches, to train a CNN forconstrained ROI localization.

Referring to FIG. 7, the trained CNN is applied at 710 during a teststage to all the pixels in an EUF 705, generating two confidence maps at715 with the same size as the EUF, one confidence map showing theprobability of a pixel being the carotid bulb, and the other confidencemap showing the probability of a pixel being the ROI. One way tolocalize the ROI is to find the center of the largest connectedcomponent within the ROI confidence map without considering the detectedlocation of the carotid bulb. However, this naive approach may fail toaccurately localize the ROI. For instance, a long-tail connectedcomponent along the far wall of the carotid artery may cause substantialROI localization error. To compound the problem, the largest connectedcomponent of the ROI confidence map may appear far from the actuallocation of the ROI, resulting in a complete detection failure. Toovercome these limitations, one embodiment constraints the ROI locationl_(roi) by the location of the carotid bulb l_(ch) as shown at 720. Forthis purpose, the embodiment determines the location of the carotid bulbas the centroid of the largest connected component within the firstconfidence map and then localizes the ROI using the following formula:

$l_{roi} = \frac{\sum\limits_{p \in {C*}}\; {{M(p)} \cdot p \cdot {I(p)}}}{\sum\limits_{p \in {C*}}\; {{M(p)} \cdot {I(p)}}}$

where M denotes the confidence map of being the ROI, C* is the largestconnected component in M that is nearest to the carotid bulb, and I(p)is an indicator function for pixel p=[p_(x), p_(y)] that is defined as:

${I(p)} = \left\{ \begin{matrix}{1,} & {{{if}\mspace{14mu} {{p - l_{cb}}}} < {1\mspace{11mu} {cm}}} \\{0,} & {otherwise}\end{matrix} \right.$

Basically, the indicator function excludes the pixels located fartherthan 1 cm from the carotid bulb location. This choice of the distancethreshold is motivated by the fact that the ROI is located within 1 cmto the right of the carotid bulb. FIG. 7 illustrates how the location ofthe carotid bulb as a contextual clue improves the accuracy of ROIlocalization.

Intima-Media Thickness Measurement

The third main step of embodiments of the invention involves measuringautomatically the thickness of the wall of the artery in each ROI usingthe machine-based artificial neural network. In one embodiment, thisinvolves measuring automatically a carotid intima-media thickness (CIMT)of a wall of a carotid artery in each ROI using the machine-basedartificial neural network. In particular, measuring the CIMT of the wallof the carotid artery comprises detecting a lumen-intima interface ofthe wall of the carotid artery, detecting a media-adventitia interfaceof the wall of the carotid artery, and then measuring a distance betweenthe lumen-intima interface and the media-adventitia interface todetermine the CIMT of the carotid artery.

To automatically measure intima-media thickness, the lumen-intima andmedia-adventitia interfaces of the carotid artery are first detectedwithin the ROI. Although the lumen-intima interface is relatively easyto detect, the detection of the media-adventitia interface is morechallenging, because of the faint image gradients around its boundary.One embodiment formulates this interface segmentation problem as a3-class classification task where the goal is to classify each pixelwithin the ROI into 3 categories: 1) a pixel on the lumen-intimainterface, 2) a pixel on the media-adventitia interface, and 3) abackground pixel.

One embodiment of the invention uses a 3-way CNN to segment thelumen-intima and media-adventitia interfaces. To train the CNN, imagepatches are collected from the lumen-intima interface andmedia-adventitia interface, as well as from other random locations farfrom the desired interfaces. FIG. 8 shows how the training patches arecollected at 805 from one ROI 810. In one embodiment, data augmentationis not performed for positive patches because ROIs of 92×60 pixels allowfor collecting a large number of patches around the lumen-intima andmedia-adventitia interfaces. Furthermore, given the relatively smalldistance between the two interfaces, translation-based data augmentationwould inject a large amount of label noise in the training set, whichwould negatively impact the convergence and the overall performance ofthe CNN. When the patches are collected, a stratified training set isformed, according to one embodiment, with approximately 380,000 patches,to train a 3-way CNN for interface segmentation.

FIG. 9 illustrates how a system according to an embodiment of theinvention measures intima-media thickness in a test ROI 900. 900(a)shows test region of interest. At 900(b), a trained CNN generates aconfidence map where the line 905 and the line 910 indicate thelikelihood of lumen-intima interface and a media-adventitia interface,respectively. 900(c) illustrates the thick probability band around eachinterface being thinned by selecting the largest probability for eachinterface in each column, and 900(d) illustrates the step-likeboundaries being refined through 2 open snakes. At 900(e), a groundtruth is made as the consensus of two experts.

The trained CNN is applied to a given test ROI in a convolutionalmanner, generating two confidence maps with the same size as the ROI.The first confidence map shows the probability of a pixel being on thelumen-intima interface; the second confidence map shows the probabilityof a pixel being on the media-adventitia interface. The two confidencemaps are shown in FIG. 9 where the lines 905 and 910 indicate thelikelihood of being the lumen-intima interface and the media-adventitiainterface, respectively. A relatively thick high-probability band isapparent along each interface, which hinders the accurate measurement ofintima-media thickness. To thin the detected interfaces, the confidencemap is scanned, column-by-column, searching for the rows with themaximum response for each of the two interfaces. By doing so, oneembodiment obtains a 1-pixel-thick boundary with a step-like shapearound each interface as shown in FIG. 9 at 900(c). To further refinethe boundaries, according to one embodiment, two active contour models(a.k.a., snakes, as described in J. Liang, T. McInerney, and D.Terzopoulos, “United snakes,” Medical image analysis, vol. 10, no. 2,pp. 215-233, 2006) are employed, one for the lumen-intima interface andone for the media-adventitia interface. The open snakes are initializedwith the current step-like boundaries and then deform solely based onthe probability maps generated by the CNN rather than the original imagecontent. 900(d) shows the converged snakes for the test ROI. In oneembodiment, intima-media thickness is determined as the average ofvertical distance between the 2 open snakes.

Returning to FIG. 2, it shows a generalized embodiment of anillustrative system 200 via which CIMT can be determined accordingembodiments of the invention. As shown, the illustrative system 200includes a computing device 210 and an ultrasound imaging device 220.The system further includes an electrocardiogram (ECG) monitoring device280. Computing device 210 can be any suitable computing device forproviding access to the video frames from device 220 and ECG signal fromdevice 280, such as a processor, a computer, a data processing device,or a combination of such devices. For example, embodiments of theinvention can be distributed into multiple backend components andmultiple frontend components or interfaces. In a more particularexample, backend components, such as data collection and datadistribution can be performed on ultrasound imaging device 220.Similarly, the graphical user interfaces displayed by the system, suchas an interface for displaying ultrasound images, ECG images, andmeasuring carotid intima-media thickness, can be distributed by one ormore computing devices 210.

Ultrasound imaging device 220 can be any suitable imaging device, suchas a high resolution B-mode ultrasound imaging device. Alternatively oradditionally, any suitable imaging device (e.g., x-ray imaging device,magnetic resonance imaging device, etc.) can be connected to thecomputing device 210 that is executing image interpretation applicationcode.

More particularly, for example, computing device 210 can be any of ageneral-purpose device such as a computer or a special purpose devicesuch as a client, a server, etc. Any of these general or special purposedevices can include any suitable components such as a processor (whichcan be a microprocessor, digital signal processor, a controller, etc.),memory, communication interfaces, display controllers, input devices,etc. For example, client 210 can be implemented as a personal computer,a tablet computing device, a personal data assistant (PDA), a portableemail device, a multimedia terminal, a mobile telephone, a gamingdevice, a set-top box, a television, etc.

In some embodiments, any suitable computer readable media can be usedfor storing instructions for performing the processes described herein,can be used to determine carotid intima-media thickness, etc. Forexample, in some embodiments, computer readable media can be transitoryor non-transitory. For example, non-transitory computer readable mediacan include media such as magnetic media (such as hard disks, floppydisks, etc.), optical media (such as compact discs, digital video discs,Blu-ray discs, etc.), semiconductor media (such as flash memory,electrically programmable read only memory (EPROM), electricallyerasable programmable read only memory (EEPROM), etc.), any suitablemedia that is not fleeting or devoid of any semblance of permanenceduring transmission, and/or any suitable tangible media. As anotherexample, transitory computer readable media can include signals onnetworks, in wires, conductors, optical fibers, circuits, or anysuitable media that is fleeting and devoid of any semblance ofpermanence during transmission, and/or any suitable intangible media.

Referring back to FIG. 2, communications link 230 (and other linksdescribed herein) may be any communications links suitable forcommunicating data between computing device 210, ultrasound imagingdevice 220, and ECG monitoring device 280, such as network links,dial-up links, wireless links, hard-wired links, any other suitablecommunications links, or a combination of such links. Computing device210 enables a user to access features of embodiments of the invention.Computing device 210 may be personal computers, laptop computers,mainframe computers, dumb terminals, data displays, Internet browsers,personal digital assistants (“PDAs”), two-way pagers, wirelessterminals, portable telephones, any other suitable access device, or anycombination of such devices. Computing device 210, ECG monitoring device280, and ultrasound imaging device 220 may be located at any suitablelocation. In one embodiment, computing device 210, ECG monitoring device280, and ultrasound imaging device 220 may be located within anorganization. Alternatively, computing device 210 and ultrasound imagingdevice 220 may be distributed between multiple organizations.

It should also be noted that computing device 210 can include processor240, storage device/memory 250, input device 260, and display 270, whichmay be interconnected. In some embodiments, memory 250 contains astorage device for storing a computer program for controlling processor240.

Processor 240 uses the computer program to present on display device 270the image interpretation and the data received through communicationslink 230 and commands and values transmitted by a user of computingdevice 210. It should also be noted that data received throughcommunications link 230 or any other communications links may bereceived from any suitable source. Input device 260 may be a computerkeyboard, a mouse, a cursor-controller, dial, switchbank, lever, or anyother suitable input. Alternatively, input device 260 may be a finger orstylus used on a touch screen display 270.

Some embodiments may include an application program interface (notshown), or alternatively, the program code may be resident in the memoryof computing device 210. In another suitable embodiment, the onlydistribution to computing device 210 may be a graphical user interface(“GUI”) which allows a user to interact with the system resident at, forexample, another computing device.

One embodiment may include client-side software, hardware, or both. Forexample, an embodiment may encompass one or more Web-pages or Web-pageportions (e.g., via any suitable encoding, such as Hyper-Text MarkupLanguage (“HTML”), Dynamic Hyper-Text Markup Language (“DHTML”),Extensible Markup Language (“XML”), Java Server Pages (“JSP”), ActiveServer Pages (“ASP”), Cold Fusion, or any other suitable approaches).

Although the invention has been described and illustrated in theforegoing illustrative embodiments, it is understood that the presentdisclosure has been made only by way of example, and that numerouschanges in the details of implementation of the invention can be madewithout departing from the spirit and scope of the invention, which isonly limited by the claims which follow. Features of the disclosedembodiments can be combined and rearranged in various ways.

1. A method for automatically determining a thickness of a wall of anartery of a subject, comprising: obtaining an ultrasound video of thewall of the artery of the subject, the ultrasound video comprising aplurality of frames of video; obtaining a correspondingelectrocardiogram (ECG) signal from the subject; selecting a subset ofthe plurality of frames of the video based on the corresponding (ECG)signal; locating automatically a region of interest (ROI) in each frameof the subset of the plurality of frames of the video using amachine-based artificial neural network; and measuring automatically athickness of the wall of the artery in each ROI using the machine-basedartificial neural network.
 2. The method of claim 1, wherein selectingthe subset of the plurality of frames of the ultrasound video based onthe corresponding ECG signal comprises selecting a plurality ofend-diastolic ultrasound frames (EUFs) based on corresponding R-waves ina QRS complex of the ECG signal.
 3. The method of claim 1, whereinselecting the subset of the plurality of frames of the ultrasound videobased on a corresponding electrocardiogram (ECG) signal comprisesautomatically selecting the subset of the plurality of frames of theultrasound video based on a corresponding electrocardiogram (ECG) signalimage displayed in the ultrasound video, using the machine-basedartificial neural network.
 4. The method of claim 3, wherein themachine-based artificial neural network is a convolutional neuralnetwork.
 5. The method of claim 1, wherein locating automatically theROI in each frame of the subset of the plurality of frames of the videousing a machine-based artificial neural network comprises: estimating,simultaneously, a location of the ROI in each frame of the subset of theplurality of frames of the video, and a location of a well-known area ofthe artery of the subject as a contextual constraint; and refining theestimated location of the ROI given the estimated location of thewell-known area of the artery.
 6. The method of claim 5, wherein thelocation of the well-known area of the artery comprises the location ofa carotid bulb of a subject's common carotid artery (CCA) and whereinthe estimated location of the ROI is approximately 1 centimeter from thecarotid bulb on a far wall of the subject's CCA.
 7. The method of claim1, wherein measuring automatically the thickness of the wall of theartery in each ROI using the machine-based artificial neural networkcomprises measuring automatically a carotid intima-media thickness(CIMT) of a wall of a carotid artery in each ROI using the machine-basedartificial neural network, and wherein measuring automatically the CIMTof the wall of the carotid artery comprises: detecting a lumen-intimainterface of the wall of the carotid artery and a media-adventitiainterface of the wall of the carotid artery; and measuring a distancebetween the lumen-intima interface and the media-adventitia interface todetermine the CIMT of the carotid artery.
 8. A system for automaticallydetermining a thickness of a wall of an artery of a subject, the systemcomprising: an ECG monitoring device that captures an electrocardiogram(ECG) signal from the subject; an ultrasound video imaging device,coupled to the ECG monitoring device, that receives the ECG signal fromthe ECG monitoring device, and captures a corresponding ultrasound videoof the wall of the artery of the subject, the ultrasound video imagingdevice producing a plurality of frames of video comprising theultrasound video of the wall of the artery of the subject and an imageof the ECG signal; a processor coupled to the ultrasound video imagingdevice, the processor configured to: select a subset of the plurality offrames of the ultrasound video based on the image of the (ECG) signal;locate automatically a region of interest (ROI) in each frame of thesubset of the plurality of frames of the video using a machine-basedartificial neural network; and measure automatically a thickness of thewall of the artery in each ROI using the machine-based artificial neuralnetwork.
 9. The system of claim 8, wherein the processor configured toselect the subset of the plurality of frames of the ultrasound videobased on the image of the ECG signal comprises the processor configuredto select a plurality of end-diastolic ultrasound frames (EUFs) based ona corresponding R-waves in a QRS complex in the image of the ECG signal,using the machine-based artificial neural network.
 10. The system ofclaim 9, wherein the machine-based artificial neural network is aconvolutional neural network.
 11. The system of claim 8, wherein theprocessor configured to locate automatically the ROI in each frame ofthe subset of the plurality of frames of the video using themachine-based artificial neural network comprises the processorconfigured to estimate a location of the ROI in each frame of the subsetof the plurality of frames of the video, and a location of a well-knownarea of the artery of the subject as a contextual constraint; and refinethe estimated location of the ROI given the estimated location of thewell-known area of the artery.
 12. The system of claim 11, wherein thelocation of the well-known area of the artery comprises the location ofa carotid bulb of a subject's common carotid artery (CCA) and whereinthe estimated location of the ROI is approximately 1 centimeter from thecarotid bulb on a far wall of the subject's CCA.
 13. The system of claim8, wherein the processor configured to measure automatically thethickness of the wall of the artery in each ROI using the machine-basedartificial neural network comprises the processor to measureautomatically a carotid intima-media thickness (CIMT) of a wall of acarotid artery in each ROI using the machine-based artificial neuralnetwork, and wherein to measure automatically the CIMT of the wall ofthe carotid artery further comprises the processor configured to: detecta lumen-intima interface of the wall of the carotid artery and amedia-adventitia interface of the wall of the carotid artery; andmeasure a distance between the lumen-intima interface and themedia-adventitia interface to determine the CIMT of the carotid artery.14. A non-transitory computer-readable medium containingcomputer-executable instructions that, when executed by a processor,cause the processor to determine a thickness of a wall of an artery of asubject, according to a method comprising: obtaining an ultrasound videoof the wall of the artery of the subject, the ultrasound videocomprising a plurality of frames of video; obtaining a correspondingelectrocardiogram (ECG) signal from the subject; selecting a subset ofthe plurality of frames of the video based on the corresponding (ECG)signal; locating automatically a region of interest (ROI) in each frameof the subset of the plurality of frames of the video using amachine-based artificial neural network; and measuring automatically athickness of the wall of the artery in each ROI using the machine-basedartificial neural network.
 15. The non-transitory computer-readablemedium of claim 14 wherein selecting the subset of the plurality offrames of the ultrasound video based on the corresponding ECG signalcomprises selecting a plurality of end-diastolic ultrasound frames(EUFs) based on corresponding R-waves in a QRS complex of the ECGsignal.
 16. The non-transitory computer-readable medium of claim 14,wherein selecting the subset of the plurality of frames of theultrasound video based on a corresponding electrocardiogram (ECG) signalcomprises automatically selecting the subset of the plurality of framesof the ultrasound video based on a corresponding electrocardiogram (ECG)signal image displayed in the ultrasound video, using the machine-basedartificial neural network.
 17. The non-transitory computer-readablemedium of claim 16, wherein the machine-based artificial neural networkis a convolutional neural network.
 18. The non-transitorycomputer-readable medium of claim 14, wherein locating automatically theROI in each frame of the subset of the plurality of frames of the videousing a machine-based artificial neural network comprises: estimating,simultaneously, a location of the ROI in each frame of the subset of theplurality of frames of the video, and a location of a well-known area ofthe artery of the subject as a contextual constraint; and refining theestimated location of the ROI given the estimated location of thewell-known area of the artery.
 19. The non-transitory computer-readablemedium of claim 18, wherein the location of the well-known area of theartery comprises the location of a carotid bulb of a subject's commoncarotid artery (CCA) and wherein the estimated location of the ROI isapproximately 1 centimeter from the carotid bulb on a far wall of thesubject's CCA.
 20. The non-transitory computer-readable medium of claim14, wherein measuring automatically the thickness of the wall of theartery in each ROI using the machine-based artificial neural networkcomprises measuring automatically a carotid intima-media thickness(CIMT) of a wall of a carotid artery in each ROI using the machine-basedartificial neural network, and wherein measuring automatically the CIMTof the wall of the carotid artery comprises: detecting a lumen-intimainterface of the wall of the carotid artery and a media-adventitiainterface of the wall of the carotid artery; and measuring a distancebetween the lumen-intima interface and the media-adventitia interface todetermine the CIMT of the carotid artery.
 21. A method for determining athickness of a wall of an artery of a subject, comprising: automaticallyselecting a subset of a plurality of frames of an ultrasound video ofthe wall of the artery on the basis of a corresponding R-wave in a QRScomplex of an electrocardiogram (ECG) signal of the subject displayed inthe plurality of frames, the subset comprising a plurality ofend-diastolic ultrasound frames (EUFs), using the machine-basedartificial neural network; locating automatically a region of interest(ROI) in each frame in the subset using a machine-based artificialneural network; and measuring automatically a thickness of the wall ofthe artery in each ROI using the machine-based artificial neuralnetwork.
 22. The method of claim 21, wherein selecting the plurality ofEUFs comprises: obtaining a portion of the ECG signal that is referencedby a cardiac cycle indicator for each of the plurality of frames;determining whether the portion of the ECG signal is the correspondingR-wave in the QRS complex of the ECG signal for each of the plurality offrames; accumulating the portions of the ECG signal that correspond tothe R-wave in the QRS complex of the ECG signal and the portions of theECG signal that do not correspond to the R-wave in the QRS complex ofthe ECG signal; and classifying the portions of the ECG signal thatcorrespond to the R-wave in the QRS complex of the ECG signal into afirst category and classifying the portions of the ECG signal that donot correspond to the R-wave in the QRS complex of the ECG signal into asecond category, using the machine-based artificial neural network. 23.The method of claim 22, wherein accumulating the portions of the ECGsignal that correspond to the R-wave in the QRS complex of the ECGsignal and the portions of the ECG signal that do not correspond to theR-wave in the QRS complex of the ECG signal comprises accumulating theportions in difference images of the plurality of frames.
 24. The methodof claim 23, wherein accumulating the portions in difference images ofthe plurality of frames comprises: identifying an image sub region forthe plurality of frames in which the ECG signal is displayed; selectingthe image sub region in each of the plurality of frames; andaccumulating the portions in difference images of the plurality offrames, comprising: subtracting every consecutive pair of selected imagesub regions to create a difference image; and summing every threeneighboring difference images to create an accumulated difference image.25. The method of claim 24, further comprising: calculating a weightedcentroid for each accumulated difference image; extracting a patch ofn×n pixels centered around the weighted centroid for each accumulateddifference image; binarize the patches; label each patch as positivethat corresponds to an R-wave in the QRS complex of the ECG signal;label each patch as negative that does not correspond to an R-wave inthe QRS complex of the ECG signal.
 26. The method of claim 25, furthercomprising: supplying the patches labeled as positive and the patcheslabeled as negative to the machine-based artificial neural network totrain the machine-based artificial neural network to distinguish betweenportions of the ECG signal that correspond to the R-wave in the QRScomplex of the ECG signal and portions of the ECG signal that do notcorrespond to the R-wave in the QRS complex of the ECG signal.
 27. Themethod of claim 26, further comprising: supplying the patches to thetrained machine-based artificial neural network; calculating aprobability for each frame being an EUF as an average of probabilitiesassigned by the machine-based artificial neural network to thecorresponding patches; concatenating the probabilities for all frames toobtain a noisy probability signal; smoothing the noisy probabilitysignal using a Gaussian function; and locating a local maxima of thesmoothed probability signal to locate the EUFs.
 28. The method of claim21, wherein locating automatically a region of interest (ROI) in eachframe in the subset of the plurality of frames of the ultrasound videoof the wall of the artery, using a machine-based artificial neuralnetwork, comprises: estimating, simultaneously, the location of the ROIin each frame of the subset, and a location of a carotid bulb of thesubject's common carotid artery (CCA) as a contextual constraint; andrefining the estimated location of the ROI given the estimated locationof the ROI is approximately 1 centimeter from the carotid bulb on a farwall of the subject's CCA.
 29. The method of claim 28, furthercomprising training the machine-based artificial neural network for3-class classification to perform the estimating, simultaneously, of thelocation of the ROI in each frame of the subset, and the location of acarotid bulb of the subject's common carotid artery (CCA) as thecontextual constraint, and refining the estimated location of the ROIgiven the estimated location of the ROI is approximately 1 centimeterfrom the carotid bulb on the far wall of the subject's CCA.
 30. Themethod of claim 28, wherein estimating, simultaneously, the location ofthe ROI in each frame of the subset, and a location of a carotid bulb ofthe subject's common carotid artery (CCA) as a contextual constraint,comprises: extracting image patches from each frame in the subset;extracting training patches centered around the locations of the carotidbulb and the ROI in each image patch; extracting background patches froma grid of points sufficiently far from the locations of the carotid bulband the ROI in each image patch; labeling and supplying the imagepatches, training patches, and background patches to the machine-basedartificial neural network to train the machine-based artificial neuralnetwork.
 31. The method of claim 30, further comprising: applying thetrained machine-based artificial neural network to all pixels in an EUF;generating a first confidence map showing a probability of a pixel beinglocated in the carotid bulb; generating a second confidence map showinga probability of a pixel being located in the ROI; and locating the ROIin the frame by finding a center of a largest connected component in thesecond confidence map, as constrained by the location of the carotidbulb based on the first confidence map.
 32. The method of claim 30,further comprising: applying the trained machine-based artificial neuralnetwork to all pixels in an EUF; generating a first confidence mapshowing a probability of a pixel being located in the carotid bulb;generating a second confidence map showing a probability of a pixelbeing located in the ROI; locating the carotid bulb as a centroid of alargest connected component in the first confidence map; and locatingthe ROI in the frame, as constrained by the location of the carotid bulbbased on the first confidence map.
 33. The method of claim 21, whereinmeasuring automatically a thickness of the wall of the artery in eachROI using the machine-based artificial neural network, comprises:measuring automatically a carotid intima-media thickness (CIMT) of awall of a carotid artery in each ROI using the machine-based artificialneural network, and wherein measuring automatically the CIMT of the wallof the carotid artery comprises: detecting a lumen-intima interface ofthe wall of the carotid artery and a media-adventitia interface of thewall of the carotid artery; and measuring a distance between thelumen-intima interface and the media-adventitia interface to determinethe CIMT of the carotid artery.
 34. The method of claim 33, wherein themachine-based artificial neural network is a three-way convolutionalneural network (CNN), and wherein detecting the lumen-intima interfaceof the wall of the carotid artery and the media-adventitia interface ofthe wall of the carotid artery comprises the three-way CNN segmentingthe lumen-intima interface of the wall of the carotid artery and themedia-adventitia interface of the wall of the carotid artery.
 35. Themethod of claim 24, further comprising training the three-way CNN,comprising: collecting image patches from the lumen-intima interface andthe media-adventitia interface; collecting image patches from randomlocations far from the lumen-intima interface and the media-adventitiainterface; creating a stratified training set of image patches from thecollected images patches of the lumen-intima interface and themedia-adventitia interface and from the collected images patches of therandom locations; and labeling and supplying the training set to thethree-way CNN to train the three-way CNN.
 36. The method of claim 35,further comprising: generating a first confidence map with the three-wayCNN that shows the probability of a pixel being on the lumen-intimainterface; generating a second confidence map with the three-way CNNthat shows the probability of a pixel being on the media-adventitiainterface; searching each confidence map for rows with a maximumresponse for each of the lumen-intima interface and the media-adventitiainterface, thereby obtaining an n-pixel thick boundary for thelumen-intima interface and an n-pixel thick boundary for themedia-adventitia interface.
 37. The method of claim 36, furthercomprising refining the boundaries using two active contour models, thetwo active contour models creating two open snakes, wherein the refiningcomprises: initializing the two open snakes, one for each n-pixel thickboundary for the lumen-intima interface and the media-adventitiainterface; deforming the two open snakes based on a probability mapgenerated by the three-way CNN; and determining intima-media thicknessas an average of a vertical distance between the two open snakes.